Methods and systems for transmitting error correction packets

ABSTRACT

The present invention discloses methods and systems for managing an error correction mode at a first communications router. The first communication router transmits data packets to a second communications router and stores the first data packet in a local storage medium. When a delay inquiry message is received from the second communications router, the first communications router activates the error correction mode. When the error correction mode is activated, the first data packet is retransmitted to the second communications router and an error correction packet corresponding to the first data packet is also transmitted. When a back-to-normal message is received from the second communications router, the first communications router deactivates the error correction mode. The back-to-normal message indicates that the first communications router no longer needs to be in error correction mode.

RELATED APPLICATIONS

The present application is a non-provisional Continuation applicationwhich claims the benefits of and is based on U.S. application Ser. No.15/295,550, titled “METHODS AND SYSTEMS FOR MANAGING ERROR CORRECTIONMODE”, filled on Oct. 17, 2016, which is a non-provisional Continuationapplication which claims the benefits of and is based on U.S.application Ser. No. 14/582,642, now U.S. Pat. No. 9,473,274, titled“METHODS AND SYSTEMS FOR TRANSMITTING DATA THROUGH AN AGGREGATEDCONNECTION”, filed on Dec. 24, 2014, which is a non-provisionalContinuation-in-part application which claims the benefits of and isbased on U.S. application Ser. No. 12/646,774, now U.S. Pat. No.9,019,827, titled “THROUGHPUT OPTIMIZATION FOR BONDED VARIABLE BANDWIDTHCONNECTIONS”, filed on Dec. 23, 2019, the disclosures of which arehereby incorporated by specific reference thereto.

TECHNICAL FIELD

The present invention relates in general to the field of computernetworks. More particularly, the present invention discloses methods andsystems for transmitting data packets frown a first communicationsrouter to a second communications router through an aggregatedconnection comprising a plurality of tunnels.

BACKGROUND ART

A multi Wide Area Network (WAN) Site-to-Site VPN router is a router thatsupports aggregating the bandwidth of multiple interconnections, e.g.,WAN connections for accessing one or more remote private networks. Insome implementations each TCP/IP session is routed to only one WAN. Inthis configuration, a single TCP file transfer session can only utilizethe bandwidth of one WAN connection on each end. For example, in asession based site-to-site virtual private network (VPN) connection. VPNtraffic is routed to multiple WAN connections between two sites (e.g.,sites A and B).

In one implementation, M.times.N tunnels are initially formed betweenthe WAN connections where M and N are the number of WAN networkconnections of site A and site B, respectively. Application TCP/IPsessions are then routed over the different tunnels. It is notable,however, that while a session based site-to-site VPN is able to utilizedifferent tunnels for different sessions, a single download session inthis type of connection is only able to utilize one tunnel.

In wireless communications, quality of packet transmission may beunpredictable, and packet drop rate may change frequently. This mayreduce the quality of the overall packet transmission. Even if thebandwidth limit of each tunnel is high, the packet drop rate may notimprove. A solution is required for utilizing multiple tunnels forincreasing the probability of successfully transmitting data, which maybe achieved by transmitting data packets more than once and also bytransmitting packets that can be used for retrieving the original datapackets.

DISCLOSURE OF INVENTION SUMMARY OF INVENTION

The present invention discloses methods and systems for managing anerror correction mode at a first communications router. The firstcommunication router transmits data packets to a second communicationsrouter. The first data packet may be received at the first communicationrouter from a first host through a local area network (LAN) interface.The first data packet is destined to a second host, wherein the secondhost is reachable through lie second communications router. The firstcommunications router stores the first data packet in a local storagemedium. When a delay inquiry message is received from the secondcommunications router, the first communications router activates theerror correction mode. The delay inquiry message comprises a globalsequence number of the first data packet and indicates that the firstdata packet has not been received by the second communications routersuccessfully. When the error correction mode is activated, the firstdata packet is retransmitted to the second communications router and anerror correction packet corresponding to the first data packet is alsotransmitted. When a back-to-normal message is received from the secondcommunications router, the first communications router deactivates theerror correction mode. The back-to-normal message indicates that thefirst communications router no longer needs to be in error-correctionmode.

According to one of the embodiments, the error correction is acombination of a plurality of packets, including the first data packet.The error correction packet may also be a forward error correction (FEC)packet, automatic repeat request (ARQ) packet, or a parity packet.According to one of the embodiments, the delay inquiry message is aninternet protocol (IP) packet, wherein the IP packet comprises a header,an other information field, and a payload. The global sequence number ofthe first data packet is in the payload and the other information fieldcomprises information that the IP packet is a delay inquiry message, andsession information of the first data packet. When a plurality of datapackets have not been received by the second communications router, thedelay inquiry message comprises global sequence numbers of the pluralityof data packets.

According to one of the embodiments, the first data packet istransmitted through an aggregated connection established between thefirst communications router and the second communications router;wherein the aggregated connection comprises a plurality of end-to-endconnections. The plurality of end-to-end connections comprises a first,second and third end-to-end connection. The first data packet istransmitted through the first end-to-end connection, the first datapacket is retransmitted through the second end-to-end connection, andthe error correction packet is transmitted through the third end-to-endconnection.

According to one of the embodiments, the local non-transitory storagemedium is a cache memory. The cache memory is capable of storing atleast twenty seconds of data transmission.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1A illustrates an overall system for optimizing throughput ofmultiple variable bandwidth connections in accordance with an embodimentof the present invention;

FIG. 1B illustrates a network environment according to variousembodiments of the present invention;

FIG. 1C illustrates system 100 adapted according to embodimentsconfigured to optimize the throughput of bonded multiple variablebandwidth connections;

FIG. 2A illustrates a flowchart depicting a method for increasingthroughput of a bonded connection in accordance with an embodiment ofthe present invention;

FIG. 2B illustrates a flowchart depicting a method for increasingthroughput of a bonded connection in accordance with an embodiment ofthe present invention;

FIG. 3 is an example embodiment illustrating the type of informationwhich may be encapsulated in a transmitted IP packet in accordance withan embodiment of the present invention;

FIG. 4A is an example embodiment illustrating the type of informationwhich may be encapsulated in a feedback packet in accordance with anembodiment of the present invention;

FIG. 4B is a chart that illustrates possible values for the fields ofthe feedback packet of FIG. 4A;

FIG. 5 depicts a block diagram of a processing system which is adaptedto implement the present invention;

FIG. 6 illustrates a network envirot according to various embodiments ofthe present invention;

FIG. 7 is a sequence diagram illustrating how data is transferredbetween hosts according to one of the embodiments of the presentinvention;

FIG. 8 is a sequence diagram illustrating how data is transferredbetween hosts according to one of the embodiments of the presentinvention;

FIG. 9 is a flowchart illustrating a process carried out at a firstcommunications router according to one of the embodiments of the presentinvention;

FIG. 10 is a flowchart illustrating a process carried out at a secondcommunications router according to one of the embodiments of the presentinvention;

FIG. 11 illustrates the structure of a delay inquiry message accordingto one of the embodiments of the present invention.

DETAILED DESCRIPTION

The ensuing description provides preferred exemplary embodiments(s)only, and is not intended to limit the scope, applicability orconfiguration of the invention. Rather, the ensuing description of thepreferred exemplary embodiment(s) will provide those skilled in the artwith an enabling description for implementing a preferred exemplaryembodiment of the invention. It being understood that various changesmay be made in the function and arrangement of elements withoutdeparting from the spirit and scope of the invention as set forth in theappended claims.

Specific details are given in the following description to provide athorough understanding of the embodiments. However, it will beunderstood by one of ordinary skill in the art that the embodiments maybe practiced without these specific details. For example, circuits maybe shown in block diagrams in order not to obscure the embodiments inunnecessary detail. In other instances, well-known circuits, processes,algorithms, structures, and techniques may be shown without unnecessarydetail in order to avoid obscuring the embodiments.

Also, it is noted that the embodiments may be described as a processwhich is depicted as a flowchart, a flow diagram, a data flow diagram, astructure diagram, or a block diagram. Although a flowchart may describethe operations as a sequential process, many of the operations can beperformed in parallel or concurrently. In addition, the order of theoperations may be re-arranged. A process is terminated when itsoperations are completed, but could have additional steps not includedin the figure. A process may correspond to a method, a function, aprocedure, a subroutine, a subprogram, etc. When a process correspondsto a function, its termination corresponds to a return of the functionto the calling function or the main function.

Embodiments, or portions thereof, may be embodied in programinstructions operable upon a processing nun for performing functions andoperations as described herein. The program instructions making up thevarious embodiments may be stored in a non-transitory storage medium.

The program instructions making up the various embodiments may be storedin a storage medium. Moreover, as disclosed herein, the term“non-transitory storage medium” may represent one or more devices forstoring data, including read only memory (ROM), programmable read-onlymemory (PROM), erasable programmable read-only memory (EPROM), randomaccess memory (RAM), magnetic RAM, core memory, floppy disk, flexibledisk, hard disk, magnetic tape, CD-ROM, flash memory devices, a memorycard and/or other machine readable mediums for storing information. Theterm “machine-readable medium” includes, but is not limited to portableor fixed storage devices, optical storage mediums, magnetic mediums,memory chips or cartridges, wireless channels and various other mediumscapable of storing, containing or carrying instruction(s) and/or data. Amachine-readable medium can be realized by virtualization, and can be avirtual machine readable medium including a virtual machine readablemedium in a cloud-based instance.

The term computer-readable medium, main memory, or secondary storage, asused herein refers to any medium that participates in providinginstructions to a processing unit for execution. The computer-readablemedium is just one example of a machine-readable medium, which may carryinstructions for implementing any of the methods and/or techniquesdescribed herein. Such a medium may take many forms, including but notlimited to, non-volatile media, volatile media, and transmission media.Non-volatile media includes, for example, optical or magnetic disks.Volatile media includes dynamic memory. Transmission media includescoaxial cables, copper wire and fiber optics. Transmission media canalso take the form of acoustic or light waves, such as those generateddining radio-wave and infra-red data communications.

A volatile storage may be used for storing temporary variables or otherintermediate information during execution of instructions by aprocessing unit. A non-volatile storage or static storage may be usedfor storing static information and instructions for processor, as wellas various system configuration parameters.

The storage medium may include a number of software modules that may beimplemented as software code to be executed by the processing unit usingany suitable computer instruction type. The software code may be storedas a series of instructions or commands, or as a program in the storagemedium.

Various forms of computer readable media may be involved in carrying oneor more sequences of one or more instructions to the processor forexecution. For example, the instructions may initially be carried on amagnetic disk from a remote computer. Alternatively, a remote computercan load the instructions into its dynamic memory and send theinstructions to the system that runs the one or more sequences of one ormore instructions.

A processing unit may be a microprocessor, a microcontroller, a digitalsignal processor (DSP), any combination of those devices, or any othercircuitry configured to process information.

A processing unit executes program instructions or code segments forimplementing embodiments of the present invention. Furthermore,embodiments may be implemented by hardware, software, firmware,middleware, microcode, hardware description languages, or anycombination thereof. When implemented in software, firmware, middlewareor microcode, the program instructions to perform the necessary tasksmay be stored in a computer readable storage medium. A processingunit(s) can be realized by virtualization, and can be a virtualprocessing unit(s) including a virtual processing unit in a cloud-basedinstance.

Embodiments of the present invention are related to the use of acomputer system for implementing the techniques described herein. In anembodiment, the inventive processing units may reside on a machine suchas a computer platform. According to one embodiment of the invention,the techniques described herein are performed by computer system inresponse to the processing unit executing one or more sequences of oneor more instructions contained in the volatile memory. Such instructionsmay be read into the volatile memory from another computer-readablemedium. Execution of the sequences of instructions contained in thevolatile memory causes the processing unit to perform the process stepsdescribed herein. In alternative embodiments, hard-wired circuitry maybe used in place of or in combination with software instructions toimplement the invention. Thus, embodiments of the invention are notlimited to any specific combination of hardware circuitry and software.

A code segment, such as program instructions, may represent a procedure,a function, a subprogram, a program, a routine, a subroutine, a module,a software package, a class, or any combination of instructions, datastructures, or program statements. A code segment may be coupled toanother code segment or a hardware circuit by passing and/or receivinginformation, data, arguments, parameters, or memory contents.Information, arguments, parameters, data, etc. may be passed, forwarded,or transmitted via any suitable means including memory sharing, messagepassing, token passing, network transmission, etc.

Alternatively, hardwired circuitry may be used in place of, or incombination with, software instructions to implement processesconsistent with the principles of the invention. Thus, implementationsconsistent with principles of the invention are not limited to anyspecific combination of hardware circuitry and software.

A network interface that may be provided by a node is an Ethernetinterface, a frame relay interface, a fibre optic interface, a cableinterface, a DSL interface, a token ring interface, a serial businterface, a universal serial bus (USB) interface, Firewire interface,Peripheral Component Interconnect (PCI) interface, etc.

A network interface may be implemented by a standalone electroniccomponent or may be integrated with other electronic components. Anetwork interface may have no network connection or at least one networkconnection depending on the configuration. A network interface may be anEthernet interface, a frame relay interface, a fibre optic interface, acable interface, a Digital Subscriber Line (DSL) interface, a token ringinterface, a serial bus interface, a universal serial bus (USB)interface, Firewire interface, Peripheral Component Interconnect (PCI)interface, cellular network interface, etc.

A network interface may connect to a wired or wireless access network.An access network may carry one or more network protocol data. A wiredaccess network may be implemented using Ethernet, fiber optic, cable,DSL, frame relay, token ring, serial bus, USB, Firewire, PCI, or anymaterial that can pass information. An wireless access network may beimplemented using infra-red, High-Speed Packet Access (HSPA), HSPA+,Long Term Evolution (LTE), WiMax, General, packet radio service (GPRS),Global System for Mobile Communications (GSM), Enhanced Data rates forGSM Evolution (EDGE), Code division multiple access (CDMA), WiFi,CDMA2000, Wideband CDMA (WCDMA), Time Division CDMA (TD-SCDMA),BLUETOOTH, WiBRO, Evolution-Data Optimized (EV-DO); Digital Enhanced.Cordless Telecommunications (DECT); Digital AMPS (IS-136/TDMA);Integrated Digital Enhanced (iDEN) or any other wireless technologies.For example, a network interface may be used as a local area network(LAN) interface or a wide area network (WAN) interface.

Embodiments, or portions thereof, may be embodied in a computer datasignal, which may be in any suitable form for communication over atransmission medium such that it is readable for execution by afunctional device (e.g., processing unit) for performing the operationsdescribed herein. The computer data signal may include any binarydigital electronic signal that can propagate over a transmission mediumsuch as electronic network channels, optical fibers, air,electromagnetic media, radio frequency (RF) links, and the like, andthus the data signal may be in the form of an electrical signal, opticalsignal, radio frequency or other wireless communication signal, etc. Thecode segments may, in certain embodiments, be downloaded via computernetworks such as the Internet, an intranet, LAN, metropolitan areanetwork (MAN), wide area network (WAN), the public switched telephonenetwork (PSTN), a satellite communication system, a cable transmissionsystem, and/or the like.

FIG. 1A illustrates system 100 adapted according to embodimentsconfigured to optimize the throughput of bonded multiple variablebandwidth connections by adjusting a tunnel bandwidth weighting schemaduring a data transfer session. System 100 includes multiple sites 102and 104, which each comprise at least one network node. A network nodemay be referred to as a communications router. However the scope of theinvention is not limited to communications routers, such that theinvention can be carried out at gateways, routers, servers, or any othertypes of network nodes. For simplicity, FIG. 1A illustrates that sites102 and 104 comprise communications router 106 and 108 respectively.Communication routers 106 and 108 may be embodied as multi WAN routerswhich support aggregating the bandwidth of multiple Internetconnections. Communications routers 106 and 108 are connected overnetwork 110. Network 110 may comprise a LAN, MAN, WAN, wireless network,the PSTN, the Internet, an intranet, an extranet, etc.

Site 102 and router 106 may comprise M connections 112, and site 104 androuter 108 may comprise N connections 114. Connections 112 and 114 aredata connections for communicating information within network 110between sites 102 and 104. In the illustrated embodiment, M is equal to3 and N is equal to 2; however, these values may vary according todesired routers and configurations. Connections 112 and 114 may havesimilar or differing bandwidth capabilities. Further, connections 112and 114 may comprise different types of WAN connections, such as a WiFi,cable, DSL, T1, 3G, 4G, satellite connections, and the like. It is alsonoted that site 102 and site 104 may be thought of as both a sender orreceiver, and discussions regarding the functionality of either site maybe implemented on the other site. In other words, system 100 may beimplemented as a symmetrical network.

FIG. 1B illustrates a network environment according to one of theembodiments of the present invention. Tunnels 103A, 103B and 103C areestablished between communications router 106 and communications router108. Tunnels 103A, 103B and 103C may be bonded to form an aggregatedconnection.

Communications routers 106 and 108 may have a plurality of networkinterfaces according to one of the embodiments. Communications router106 establishes tunnels 103A, 103B, and 103C via one or more of itsplurality of network interfaces with one or more network interfaces ofcommunications router 108.

Communication device 106 and 108 may work as a gateway, a router, aswitch, an access point, a hub, a bridge, etc.

FIG. 1C illustrates system 100 adapted according to embodimentsconfigured to optimize the throughput of bonded multiple variablebandwidth connections. System 100 is similar to system 101, with theexception of M×N virtual tunnels 116. When establishing a bondedconnection between sites 102 and 104, such as by implementing a bondedsite-to-site VPN connection, M.times.N tunnels 116 may be created.Tunnels 116 correspond to a unique permutation of the networkconnections of site 102 and the network connections of site 104. Anaggregated connection may be formed between communications routers 106and 108. Tunnels 116 may be virtual tunnels.

A plurality of established tunnels 116 may be aggregated, combined orbonded together to form one aggregated connection. Those skilled in thearts would appreciate that there are myriad ways to aggregate, combine,or bond a plurality of established tunnels to forum one aggregatetunnel. An aggregated connection is perceived as one tunnel by sessionsor applications that are using it. An aggregated connection may be anend-to-end connection, a virtual private network connection orconnectionless oriented connection. For example, an aggregatedconnection may be a TCP connection or UDP connection. In anotherexample, aggregated connection is an aggregation of a plurality oftunnels, and each tunnel is linked between communications router 106 andcommunications router 108. In another example, an aggregated connectionmay be a VPN tunnel, comprising a plurality of established tunnels, andeach established tunnel is linked between communications router 106 andcommunications router 108.

FIG. 2A shows a high level flow diagram of operation of system 100depicting a method 200 for increasing throughput of a bonded connection.It should be appreciated that the particular functionality, the order ofthe functionality, etc. provided in FIG. 2 is intended to be exemplaryof operation in accordance with the concepts of the present invention.Accordingly, the concepts herein may be implemented in various waysdiffering from that of the illustrated embodiment.

At block 201 of the illustrated embodiment when establishing a bondedconnection between routers 102 and 104, such as by implementing a bondedsite-to-site VPN connection, M.times.N virtual tunnels 116 may becreated, as illustrated in FIG. 1C. Virtual tunnels 116 correspond to aunique permutation of the network connections of site 102 and thenetwork connections of site 104.

At block 202 of the illustrated embodiment, default weights for thetunnels are determined and/or assigned. To determine default weightsembodiments exchange uplink and downlink bandwidth data of connections112 and 114 between sites 102 and 104. Using this bandwidth data, adefault weight may be calculated according to the following: supposesite 102's downlink bandwidths of connections 1 to m are d1 d2, . . .dm, and site 104's uplink bandwidths of connections 1 to n are ur, U2, .. . Un; the default weight for the tunnel between site 102's connectionX and site 104's connection V may be defined as DW(x,y), whereDW(x,y)=dx,dy. Using the above method to calculate default weight, ifconnections 112-1 through 112 3 are WAN connections of a multi WANrouter with respective uplink/downlink bandwidths of 10M/6M, 8M/4M, and6M/6M, and connections 114-1 through 114-2 are WAN connections of amulti WAN router with respective uplink dot bandwidths of 7M/5M and9M/3M, the respective default weights for each tunnel will be asfollows:

TABLE 0001 For site 102 For site 104 DW(1, 1) = 6 * 7 = 42 DW(1, 1) =5 * 10 = 50 DW(1, 2) = 6 * 9 = 54 DW(1, 2) = 5 * 8 = 40 DW(2, 1) = 4 * 7= 28 DW(1, 3) = 5 * 6 = 30 DW(2, 2) = 4 * 9 = 36 DW(2, 1) = 3 * 10 = 30DW(3, 1) = 6 * 7 = 42 DW(2, 2) = 3 * 8 = 24 DW(3, 2) = 6 * 9 = 54 DW(2,3) = 3 * 6 = 18

It is noted that other ways to calculate default weight arecontemplated, and the above is simply an example of the implementationof an embodiment of the present invention. It is noted that manydifferent weighting schema may be used to define the initial bandwidthof a tunnel. For example, one may desire to only weight a tunnel in onedirection using the downlink capacity of a receiving site and the uplinkcapacity of the sending site. Any weighting scheme used to characterizecapacity of the tunnels at the establishment of the bonded connectionmay be used for the purposes of the present invention.

When packets are being routed from site 102 to site 104 according toembodiments, the packets will be distributed to the tunnels in a ratioaccording to an effective weight EW(x,y). Initially the effective weightof embodiments is set to be equal to the default weight,EW(x,y)=DW(x,y), and if the bandwidth of tunnels 116 remains unchangedfrom the initial setting, the effective weight is optimal for packetdistribution. However, if a user is downloading a file over a bondednetwork connection in a TCP session with one or more tunnels havingpacket drops, the overall throughput of the session will dropdramatically. This is in part because the packet drops will keep causingTCP retransmissions and TCP's flow control will maintain a lowerthroughput even though tunnels without packet drops are not fullyoccupied.

One effective was to increase throughput would be to avoid such packetdrops. To do so, embodiments of the present invention discern whentunnels are experiencing an increase or decrease in packet drop rates atblock 203 of the illustrated embodiment. Embodiments further function tomodify the effective weight of tunnels which are experiencing or haveexperienced changes in packet drop rates at block 204. The packet droprate information may be monitored continuously or be monitored based onspecific time periods. Once it is determined that a tunnel isexperiencing an unacceptable rate of packet drops (block 204-1), theillustrated embodiment decreases the effective weight of the tunnel atblock 204-2. In some embodiments, unacceptable may mean that the packetdrop rate is a non-zero quantity, while other embodiments may determinethat an unacceptable rate is any rate beyond a predefined threshold.Embodiments implement these decreases in stepwise fashion, in acontinuous manner, in a reduction at one time in proportion to theincrease in the packet drop rate, etc. When reductions are done in agradual manner, embodiments, may continue to monitor the tunnel in orderto optimize the amount of reduction which is implemented.

Tunnels 116 may be established or monitored by sending heartbeat packetsthrough each tunnel from either router 106 or router 108. In someembodiments when the receive end fails to receive heartbeat packets froma tunnel for a period of time, it will treat that tunnel as down and thetunnel will not be used for routing traffic. If heartbeat packets againstart being received, the tunnel may be re-established and be, weightedalong with the other tunnels. As such, in the event that all packets arebeing dropped in a tunnel and the effective weight of that tunnel isreduced to zero, embodiments may utilize heartbeat packets to monitorand reestablish a connection.

Moreover, when tunnels recover all or part of their respectivebandwidths, e.g. it is determined that the packet drop rate decreases(block 204-3), the illustrated embodiment functions to increase theeffective weight of such tunnels (block 204-4) in order to fully, ormore fully, utilize the bandwidth. Some embodiments increase theeffective weight for a tunnel using predetermined step sizes until anaccurate effective weight is regained. Other embodiments increase theeffective weight proportionate to a newly measured bandwidth which maycorrespond to a newly measured packet drop rate. Moreover, embodimentsmay increase the effective weight for a tunnel based on a predeterminedlinear or exponential scale.

After the effective weight of the tunnels are adjusted, or it isdetermined that no adjustment is needed, the weighting scheme of thesystem is updated at block 205 of the illustrated embodiment. Thisupdate may comprise storing any processed, information, using suchinformation in further processing, causing the system to take no action,etc. For example, processing performed with respect to block 205 mayoperate to average weighting schemes over a period of time, such as tomitigate error associated with highly transient anomalies. Further, theupdated, information may be used on system 100 to modify the packetdistribution of the data transfer session, as discussed with respect toFIG. 2B, System 100 may continue to implement steps 203-205 continuouslyor periodically throughout a data transfer session.

FIG. 2B illustrates an embodiment where, after weighting method 200 isimplemented, the packets are distributed based, at least in part, on themodified weight of the tunnels. Specifically, block 206 of theillustrated embodiment operates to distribute packets across the tunnelsin accordance with the weighting scheme determined by operation ofmethod 200. In some embodiments, this distribution will changethroughout a data transfer session, and therefore the steps of FIG. 2Bare shown as repeating. Some embodiments change the packet distributioneach time the system is updated at block 205. Moreover, block 205 maycause changes to be implemented periodically, in response to certaindrop rate change thresholds, etc. It should be appreciated that thedetermination of weighting, by operation of method 200 and theapplication of determined weighting to packet distribution at block 206may have different periodicity. For example, method 200 may operate toprovide updates of weighting scheme information using a relatively shortiterative cycle while the distribution of packets is altered based uponsuch weighting scheme information using a longer iterative cycle.

To monitor the bandwidth of the various tunnels 116, some embodiments ofthe present invention encapsulate each transmitted IP packet withvarious information. FIG. 3 illustrates an example embodiment showingthe type of information 300 which may be encapsulated in a transmittedIP packet. Version field 302 may contain information about the protocolversion being utilized and protocol type field 303 may contain theprotocol type of the payload packet. In general, the value of this fieldwill correspond to the Ethernet protocol type for the packet. However,additional values may be defined in other documents. Tunnel ID field 304may be a 32-bit field and may contain an identifier to identify thecurrent tunnel of the IP packet. Advanced Encryption Standard (AES)initialization sector field 306 may be a 32-bit field and may contain aninitialization vector for AES encryption. Global sequence number field308 may be a 32-bit field and may contain a sequence number which isutilized to re-sequence each of the packets for various sessions intothe proper order when they have emerged from their respective tunnels.Per tunnel sequence number field 310 may be a 32-bit field which mayrepresent a sequence number that is assigned to each packet routed to aparticular tunnel. AES encrypted payload field 312 may be utilized toconvey the payload of the IP packet. AES encryption, may be applied forhigher security of the payload in order to prevent attacks from thirdparties.

The per tunnel sequence number discussed above may be used to monitordropped packets in a tunnel. In one embodiment the router on thereceiving end calculates the packet drop rate of each tunnel, DR(x,y),every f seconds by monitoring the per tunnel sequence number of thereceived packets. DR(x,y) may be characterized as the sequence numbersmissed divided by a sequence number increase for a period f. The lengthof period f may vary, and in one embodiment f is equal to 5 seconds.

Other methods may also be used to monitor dropped packets, e.g. thesender may periodically inform the receive end how many packets it hassent, the sender sends a heartbeat packet to the receive end everyconstant period of time and the receive end can estimate the overalldrop rate by monitoring the heartbeat packets' drop rate, by acquiringdrop rate figures from physical interface/device/layer, etc.

The receive end may feedback a particular tunnel's drop rate, effectiveweight, or other bandwidth indicators, to the sending router. When thesender receives information regarding packet drops, some embodimentslower the effective weight EW(x,y) of a tunnel by EW(x,y)DR(x,y). Othermetrics may be used to modify the effective weight of a tunnel. In someembodiments, the sender may receive feedback and the effective weightmay be reduced by number that is greater than or less than the packetdrop rate. Such variances may be configured according to the particularneeds of a communication system. The above example represents a metricthat attempts to lower the effective weight of the tunnel to a weightwhich prevents further packet drops while maximizing the amount ofusable bandwidth of the tunnel. Any metric which finds this balance maybe preferred.

FIG. 4A illustrates an example embodiment of the type of information 400which may be encapsulated in a feedback packet which is sent to thetransmitting router in order to report packet drop rates or otherbandwidth related data received at the receiving end router. Type field402 may include data regarding the type of data that will be included indata 1 field 404 and data 2 field 406. Data 1 field 404 and data 2 field406 may contain any information which may be used to assist the routerin determining tunnel information with regard to the number of tunnels,bandwidth of tunnels, number of dropped packets in a tunnel, and thelike. An example of possible values of the type field 402 in the datafields 404 and 406 is shown in the chart of FIG. 4B.

The information which is encapsulated in transmitted IP packets, such asshown in FIG. 3 and FIG. 4 may also be used for packet buffering andre-sequencing. Because each tunnel's latency can be different, when twoconsecutive packets of the same TCP session are sent to a VPN peer overa bonded VPN tunnel, they may not arrive in sequence because they arerouted via two different tunnels. If the TCP session receives theout-of-sequence packets from the VPN, the TCP session will slow down dueto TCP retransmissions. Accordingly, the receive end should buffer thepackets that come too early until either the slower packets arrive oruntil an expiration time has passed. With such buffering, late packetsthat come prior to an expiration time will be forwarded to thedestination device in sequence. This buffering assists in theoptimization of end-to-end throughput.

It is noted that embodiments described herein are, at times, discussedin the context of a VPN connection. These discussions are presented inorder to show an example embodiment of a bonded connection. Theinventive concepts described in claimed herein are not limited to suchconnections. In fact, any connection where sufficient data may beobtained and exchanged in order to dynamically monitor the bandwidth ofa plurality of communication paths which are being used in a datatransfer session may be implemented with the embodiments of the presentinvention.

As discussed above, each packet may be assigned two different sequencenumbers, a global sequence number (GSN) and a per tunnel sequence number(PTSN). These numbers may be used to assist in packet buffering andre-sequencing operations. After a packet is passed to an upper layer,the receive end may update a next expected per-tunnel sequence number(NE-PTSN) and a next expected global sequence number (NE-GSN).

The following will describe one method of how a packet may be bufferedor forwarded to destination device after it is received and decrypted.

1. If the packet's GSN equals to zero, forward it to destination deviceimmediately.

2. Check if the packet's PTSN equals to the NE-PTSN. If not, dequeue(forward to destination device) in sequence all packets that have asmaller GSN than the packet's. Keep the packet unprocessed.

3. Update the NE-PTSN (i.e., set NE-PTSN to PTSN+1).

4. If the GSN is less than the NE-GSN, forward to destination device.

5. If the packet's GSN is equal to the NE-GSN, update the NE-GSN (i.e.,set NEGSN to GSN+1) and forward to destination device. Repeat updatingthe NE-GSN and dequeuing the buffer head from the butler if the head'sGSN equals to the new NE-GSN.

6. Otherwise (GSN is larger than the NE-GSN), enqueue the packet in theorder of the GSN.

7. If a packet has been in the queue longer than a fixed amount of time,set the NEGSN to the packet's GSN+1 and dequeue in sequence the packetand all packets that have a smaller GSN than the packet's.

Therefore, the encapsulated packet information discussed in FIG. 2 andFIG. 3 may include information that optimizes overall throughput of thedata transmission system, such as 100, both by assisting in theoptimization of tunnel bandwidth in response to monitoring packet droprates, and by assisting in the efficient re-sequencing of receivedpackets in a data transfer session.

FIG. 5 illustrates an exemplary processor-based system 500 which may beemployed to implement the systems, devices, and methods according tocertain embodiments. Processor-based system 500 may represent thearchitecture of communications router 106 and 108. Central processingunit (CPU) 501 is coupled to system bus 502. CPU 501 may be any generalpurpose CPU, or may be a special purpose CPU designed to implement theabove teachings. The present disclosure is not restricted by thearchitecture of CPU 501 (or other components of exemplary system 500) aslong as CPU 501 (and other components of system 500) supports theinventive operations as described herein. CPU 501 may execute thevarious logical instructions described herein. For example, CPU 501 mayexecute machine-level instructions according to the exemplaryoperational flow described above in conjunction with FIG. 2. Whenexecuting instructions representative of the operational stepsillustrated in FIG. 2, CPU 501 becomes a special-purpose processor of aspecial purpose computing platform configured specifically to operateaccording to the various embodiments of the teachings described herein.

System 500 also includes random access memory (RAM) 503, which may beSRAM, DRAM, SDRAM, or the like. RAM 503 may be a secondary storage whichstores program instructions executable by CPU 501. System 500 includesread-only memory (ROM) 504 which may be PROM, EPROM, EEPROM, or thelike. RAM 503 and ROM 504 hold user and system data and programs, as arewell known in the art.

System 500 also includes input/output (I/O) adapter 505, communicationsadapter 511, user interface adapter 508, and display adapter 509. I/Oadapter 505, user interface adapter 508, and/or communications adapter511 may, in certain embodiments, enable a user to interact with system500 in, order to input information.

I/O adapter 505 connects storage device(s) 506, such as one or more ofhard drive, compact disc (CD) drive, floppy disk drive, tape drive,etc., to system 500. The storage devices are utilized in addition to RAM503 for the memory requirements associated performing the operationsdiscussed in the above embodiments. Communications adapter 511 isadapted to couple system 500 to network 512, which may enableinformation to be input to and/or output from system 500 via suchnetwork 512 (e.g., the Internet or other wide-area network, a local-areanetwork, a public or private switched telephony network, a wirelessnetwork, any combination of the foregoing). Communications adapter 511may be regarded as a network interface, and system 500 may comprise aplurality of communications adapters 511. User interface adapter 508couples user input devices, such as keyboard 513, pointing device 507,and microphone 514 and/or output devices, such as speakers) 515 tosystem 500. Display adapter 509 is driven by CPU 501 to control thedisplay on display device 510. Display adapter 509 transmitsinstructions for transforming or manipulating the state of the variousnumbers of pixels used by display device 510 to visually present thedesired information to a user. Such instructions include instructionsfor changing state from on to off setting a particular color, intensity,duration, or the like. Each such instruction makes up the renderinginstructions that control how and what is displayed on display device510.

FIG. 6 illustrates a network environment based on the networkenvironment illustrated in FIG. 1A. One of the LAN interface ofcommunications router 106 is connected to host 103 and one of the LANinterface of communications router 108 is connected to host 105. Hosts103 and 105 are in sites 102 and 104 respectively. A plurality oftunnels may be established between communications routers 106 and 108through network 110. Communications router 106 may establish tunnelswith communications router 108 through one or more WAN interfaces ofcommunications router 106 and one or more WAN interfaces ofcommunications router 108. For illustration purpose only, communicationsrouter 108 connects to the Internet 110 through two connections whilecommunications router 106 connects to the network 110 through twoconnections. Network 110 may be an interconnected network, such as theinternet.

FIG. 7 is a sequence diagram illustrating how data is transferredbetween host 105 and host 103 according, to one of the embodiments ofthe present invention. In flow 701, data-1 is transmitted from host 103to communications router 106. Data-1 is originated at host 103, and isintended to be transmitted to host 105. Data-1 can be comprised in oneor more data packets, one or more frames, or any other data unit.Whether data-1 is comprised in one data packet or more data packetsdepends on the size of data-1. If size of data-1 is such that data-1 canbe encapsulated in one data packet, then data-1 is comprised in one datapacket. If size of data-1 is too big to be encapsulated in one datapacket, data-1 may be fragmented into a plurality of parts and theplurality of parts are encapsulated in a plurality of data packetsrespectively. Communications router 106 then sends data-1 tocommunications router 108 in flow 702 through interconnected networks110. Communications router 108 forwards data-1 to host 105. When host105 receives data-1, it creates an acknowledgement packet, Ack-1, whichis destined to host 103. Ack-1 is received at communications router 108at flow 704, and forwarded to communications router 106 at flow 705.Communications router 106 then sends Ack-1 to host 103 in flow 706.Therefore host 103 now knows that data-1 reached host 105 successfully.Host 103 then transmits data-2 to host 105 through communicationsrouters 106 and 108. Data-2 is transmitted from host 103 tocommunications router 106 in 707. Communications router 106 forwardsdata-2 to communications router 108 in flow 708. Communications router108 does not receive data-2 on time, as data-2 may be dropped or may bedelayed when being transmitted to communications router 108.Communications router 108 then creates and sends a delay inquiry messagein flow 709 to inform communications router 106 that data-2 has not beenreceived yet and to check why there is a delay. When communicationsrouter 106 receives the delay inquiry message, it determines that data-2had been lost, and therefore it activates an error correction mode.Under the error correction mode, communications router 106 sendsdata-2-1 and data-2-EC in flows 710 and 711 respectively. Data-2-1 maybe the same as data-2, whereas data-2-EC comprises error correction datacorresponding to data-2. Using data-2-1 and/or data-2-EC, communicationsrouter 108 can create data-2 and send data-2 to host 105 in flow 712.When host 105 receives data-2, it creates an acknowledgement packet,Ack-2, which is destined to host 103. Ack-2 is received atcommunications router 108 at flow 713 and forwarded to communicationsrouter 106 at flow 714. Communications router 106 then sends Ack-2 tohost 103 in flow 715. Host 103 sends data-3 to host 105 throughcommunications routers 106 and 108. Data-3 is transmitted tocommunications router 106 in flow 716. Since the error correction modeis activated in communications muter 106, communications router 106sends both data-3 and data-3-1, and also and data-3-EC to communicationsrouter 108 in flows 717, 718 and 722 respectively. Communications router108 successfully receives data-3-1 and forwards data-3 to host 105 inflow 719. Host 105 then creates and sends an acknowledgement packet,Ack-3, which is destined to host 103. Ack-3 is received atcommunications router 108 at flow 720, and forwarded to communicationsrouter 106 at flow 721. Communications router 106 then sends Ack-3 tohost 103 in flow 723. As communications router 108 receives bothdata-3-1 and data-3-EC, it determines that data is no longer being lost,and therefore sends a back-to-normal message to communications router106 in flow 724. When communications muter 106 receives theback-to-normal message, it can deactivate the error correction mode.When the error correction mode is deactivated, communications router 106no longer sends error correction packets such as data-2-EC anddata-3-EC.

In some communication protocols, such as TCP, when a data transfersession ends, an end message may be sent from the sender to the receiverin order to indicate that the data transfer session has ended, i.e., nomore data packets belonging to the data transfer session will be sent.Communications router 108 may determine that data, such as data-2, isdelayed or lost, as it may not have received any end message afterreceiving data-1. Therefore, it sends the delay inquiry message in flow709 in order to request communications router 106 to resend data-2.

When data is received at communications router 106 from host 103,communications router 106 may save the data in a local storage medium,such as a cache, such that it may send copies of the data when necessarywithout requiring host 103 to resend the data. In case the data is lostwhen being transmitted to communications router 108, communicationsrouter 106 can quickly resend the data. There is no need to wait forhost 103 to resend the data or inform host 103 about the lost. Those whoare skilled in the art would appreciate that in some communicationprotocols, when host 103 is aware the host that data is lost, host 103may slow down the data transmission. Since the waiting time is reduced,communications router 108 is able to receive a retransmission of thedata from communication router 106, instead of making a retransmissionrequest to host 103.

In flow 721, communications router 106 sends out data-3-EC. Forillustration purpose, coincidentally data-3-EC is sent before Ack-3 isreceived at communications router 106. In one variant, if Ack-3 had beenreceived earlier at communications router 106, communications router 106would not have transmitted data-3-EC.

Contents of data-2-1 may be the same as contents of data-2, ascommunications router 106 resends data-2 by sending data-2-1. Data-2-ECis encapsulated in error correction packets. In one variant, the datapacket encapsulating data-2-1 is identical to the data packetencapsulating data-2 as the content of data-2-1 is the same as thecontent of data-2, source addresses of the data packets are the same anddestination addresses of the data packets are the same.

Error correction packets may be packets that are transmitted bycommunications router 106 to correct or detect any errors in datapackets that are transmitted from communications router 106 tocommunications router 108. Error correction packets include Forwarderror correction (FEC) packets or Automatic repeat request (ARQ)packets, Parity packets, or the like. In one variant, error correctionpackets may also be combinations of two or more packets. For example, ifa first packet comprises data-2 and a second packet comprises data-2-EC,a third packet may comprise data content based on an exclusive OR (XOR)operation applied to the contents of first and second packets. Usingdata-2-EC, communications router 108 may be able to retrieve data-2, andsends data-2 to host 105.

In one variant, data-2 may be fragmented into two parts. A first part ofdata-2 is encapsulated in a first data packet and a second part ofdata-2 is encapsulated in a second data packet. For illustrationpurpose, when communications router 106 sends data-2 to communicationsrouter 108 in flow 708, communications router 108 receives the firstdata packet, but the second data packet is dropped or lost.Communications router 108 then sends the delay inquiry message in flow709. Data-2-EC may comprise an error correction packet whose datacontent is based on an XOR operation applied to the contents of thefirst data packet and the second data packet, i.e., the XOR operation isapplied to the first part of data-2 and the second part of data-2. Whencommunications router 106 sends data-2-EC, communications router 108 canretrieve the second part of data-2 by using the first part of data-2 anddata-2-EC. It would be known to those skilled in the art how the secondpart of data-2 can be retrieved using the first part of data-2 anddata-2-EC, as data-2-EC is based on the first and second part of data-2.

In one variant, when communications router 106 receives the delayinquiry message and error correction mode is activated, communicationsrouter 106 may only send error correction packets, such as data-2-EC,and does not send data-2-1. In another variant, when communicationsrouter 106 receives the delay inquiry message, communications router 106may only send data-2-1 and not send data-2-EC. The benefit of sendingeither only one of data-2-1 and data-2-EC is that fewer number ofpackets are transmitted, and hence lower bandwidth is consumed. This mayhelp reduce traffic congestion. However, in some scenarios, data-2-ECalone may not help in retrieving data-2 if data-2-1 is not received. Forexample, when data-2-EC only contains a checksum or parity packets, itcan only be used for error detection. In another example, when data-2-ECis created by applying XOR operation on two packets of data-2, data-2can only be retrieved from data-2-EC if at least one of the two packetsof data-2 had been received. Thus, sending data-2-1 may be helpful, suchthat data-2 may be retrieved using data-2-1 and data-2-EC. In onevariant, as there may be some errors in data-2-1 received atcommunications router 108, data-2-EC may be used to correct the errors.

FIG. 11 illustrates the structure of a delay inquiry message accordingto one of the embodiments of the present invention. Delay inquirymessage 1100 may be sent when any data packet is dropped or lost, i.e.,there is a missing data packet. Delay inquiry message 1100 may be an IPpacket comprising IP header 1101, other information field 1104 andpayload 1105. IP header 1101 comprises a source address field 1102 anddestination address field 1101. For example, referring to FIG. 7, thedelay inquiry message sent in flow 709 is the source address is IPaddress of communications router 108 and the destination address is theIP address of communications router 106. Other information field 1104may include various information such as the nature of the IP packet,i.e. information that the TP packet is a delay inquiry message. Thisindicates to communications router 106 that the IP packet is not a datapacket, and just a management message. Communications router 106 mayprocess data packets and management messages differently. For example,communications router 106 may store data packets received fromcommunications router 108 in a cache, but communications router 106 maynot store management messages such as the delay inquiry message in thecache. Other information field 1102 may further include sessioninformation in order to indicate which session the missing packetbelongs to. Payload 1105 may contain GSN of the missing packet. Forexample, in flow 709, payload 1105 of delay inquiry message 1100 sent bycommunications router 108 to communications router 106 may contain GSNof one or more data packets corresponding to data-2 as data-2 had notreached communications router 108 successfully. When communicationsrouter 106 receives delay inquiry message 1100 from communicationsrouter 108, and payload 1105 contains GSN of one or more data packetscorresponding to data-2, communications router 106 may determine thatcommunications router 108 has not received data-2. Thereforecommunications router 106 then sends data-2-1 and data-2-EC tocommunications router 108. The scope of the invention is not limited tothe delay inquiry message being an IP packet. The delay inquiry messagemay be a short messaging service (SMS) message, a multimedia messagingservice (MMS) message, or any other type of message that can be sent bycommunications router 108 to communications router 106 for givinginformation of any missing data packets.

FIG. 8 is a sequence diagram illustrating how data is transferredbetween host 105 and host 103 according to one of the embodiments of thepresent invention. Host 103 sends data-1 to host 105 throughcommunications routers 106 and 108. Data-1 is transmitted from host 103to communications router 106 in flow 801. Communications router 106 thentransmits data-1 to communications router 108 in flow 802 andcommunications router 108 forwards data-1 to host 105 in flow 803.Communications router 106 is initially in error correction mode, andhence, after a waiting time period, it sends data-1-1 and data-1-EC inflows 806 and 807 respectively. The benefit of transmitting data-1-1 anddata-1-EC is that in case data-1 was lost, host 103 would not have toretransmit data-1. Data-1-1 and data-1-EC may help communications router108 to create and send data-1 to host 105 in case data-1 was lost inflow 802. When host 105 receives data-1, it creates an acknowledgementpacket, Ack-1, which is destined to host 103. Ack-1 is transmitted tocommunications router 108 in flow 804 and is transmitted bycommunications router 108 to communications router 106 in flow 805. Ascommunications router 108 receives both data-1 and data-1-1 fromcommunications router 106, it sends a back-to-normal message tocommunications router 106 in flow 808. After receiving theback-to-normal message to communications router 106 deactivates theerror correction mode as packets are no longer being lost. Host 103transmits data-2 which is destined to host 105. When data-2 is receivedat communications router 106 in flow 810, communications router 106forwards data-2 to communications router 108 in flow 811. Communicationsrouter 108 then forwards data-2 to host 105 in flow 812. As the errorcorrection mode has been deactivated, communications router 106 does notsend any error correction packets for data-2. When host 105 receivesdata-2. it creates an acknowledgement packet, Ack-2, which is destinedto host 103. Ack-2 is sent to communications router 108 in flow 813, andis forwarded to communications router 106 in flow 814. Communicationsmuter 106 then sends Ack-2 to host 103 in flow 815.

The waiting time period between communications router 106 sending data-1and sending data-1-1 may be adjusted or adapted to different networkingneed. The purpose of transmitting data-1-1 is to reduce the probabilityof host 103 retransmitting data-1. Host 103 may retransmit data-1 forone or both of the following reasons: (i) when host 103 determines thatdata-1 is lost because an acknowledgement has not been received, and(ii) when host 105 sends a request to host 103 to resend data-1.Therefore, in one variant, the waiting time period is preferably smallerthan both (i) a first time period between host 103 sending data-1 andhost 103 deciding to retransmit data-1, and (ii) a second time periodbetween host 103 transmitting data-1 and host 105 deciding to send arequest to host 103 to resend data-1. The first time period and thesecond time period may be defined in the transmission protocol, such asTCP protocol. As a certain delay is caused for data to travel fromcommunications router 106 to communications router 108, a third timeperiod required for a data packet to be transmitted from communicationsrouter 106 to communications router 108 may also be taken intoconsideration for setting the waiting time period. The third time periodmay vary according to network conditions. The waiting time period maythen be smaller than the first time period minus the third time period,and may also be smaller than the second time period minus the third timeperiod. The benefit of reducing the probability of host 103retransmitting data-1 is that host 103 does not slow down the datatransmission because of the packet drop. In another variant the waitingtime period is larger than a fourth time period, which is the timerequired, in general, for Ack-1 to be transmitted from communicationsrouter 108 to communications router 106. This may cause communicationsrouter 108 to wait for receiving Ack-1 for the time required, and ifAck-1 is not received within the fourth time period, data-1-1 anddata-1-EC are transmitted. The benefit of the waiting time period beinglarger than the fourth time period is that less bandwidth may beconsumed, as data-1-1 and data-1-EC is not transmitted if Ack-1 isreceived. However, the fourth time period may vary according to networkconditions. The disadvantage of the waiting time period being largerthan the fourth time period is that when the fourth time period haspassed, host 103 may retransmit data-1, and may also slow down the datatransmission. Therefore it is preferred to take into consideration thefirst, second, and third time periods as discussed above, and ignore thefourth time period. This ensures that data is transmitted successfullyand data transmission is not slowed down, even though bandwidthconsumption may be higher. In another variant, the waiting time periodis adjustable by the administrator of communication routers 106 and 108.

The difference between the sequence diagrams in FIG. 7 and FIG. 8 isthat in FIG. 7, the error correction mode is activated at communicationsrouter 106 when a delay inquiry message is received at communicationsrouter 106. Alternatively, in FIG. 8, the error correction mode may beactivated by default, or may be activated by a user or administrator ofcommunications router 106. If the error correction mode is activated bydefault, there may be traffic congestion caused by error correctionpackets and resending of data packets. This may slow down the datatransmission and consume significantly high bandwidth. Therefore, it maybe beneficial to activate the error correction mode only when a delayinquiry message is received in order to save bandwidth and reducetraffic congestion. However, having the error correction mode activatedby default may make the process of resending data packets and errorcorrection packets faster, as communications router 106 does not need towait for communications router 108 to send the delay inquiry message.This can be beneficial when communications router 106 already knows thatthe packet drop rate or the packet loss rate of a tunnel is high. Whenpacket drop rate or packet loss rate is high, communications router 106resends data packets and sends error correction packets.

In an example, when host 103 is transmitting data to host 105 throughcommunications router 106 and 108 using TCP, and many packets are beingdropped or lost, host 103 may reduce transmission rate in order toreduce packet loss and packet drop. This may make the overalltransmission much slower, even if the increase in packet drop istemporary. Additionally, when host 103 determines that a packet has beendropped or lost, it will resend the packet. The overall transmission maybe faster when communications router 106 resends packets, compared towhen host 103 resends packets. In order to avoid host 103 reducingtransmission rate and resending packets, communications router 106 sendsmore than one copy of the same packet, for example data-2 and data-2-1,and may also send error correction packets, for example data-2-EC. Whencommunications router 106 sends more than one copy of the same packet,redundancy is higher and it is more likely that the packet will bereceived at host 105. Therefore, acknowledgement packets may be sentfrom host 105 and received at host 103 on time, and host 103 would notneed to reduce the transmission rate or resend packets.

In a preferred embodiment, copies of the same packet are transmittedthrough different tunnels of an aggregated connection, if possible. Forexample, referring to FIG. 7, if data-2 is transmitted in flow 708through tunnel 103A, data-2-1 is transmitted in flow 710 through tunnel103B and data-2-EC is transmitted in flow 711 through tunnel 103C. Thismay allow data-2, data-2-1 and data-2-EC to be transmitted substantiallyat the same time. Furthermore, if tunnels 103A, 103B and 103C areestablished using networks provided by different service providers, thechance of at least one of data-2, data-2-1 and data-2-EC reachingcommunications router 108 successfully is higher. This is because thenetwork quality of the tunnels may differ. If the network quality oftunnel 103A is not satisfactory or tunnel 103A is broken or tunnel 103Ais experiencing lots of packets drop, data-2 may not reachcommunications router 108 successfully. If data-2 is sent through tunnel103A, data-2-1 may experience the same network problem as data-2 hasjust experienced. Therefore, it is preferred to send data-2-1 through adifferent tunnel, such as tunnel 103B, which may experience betternetwork performance as the path, route or connection used by tunnel 103Bmay be different from the path, route or connection used by tunnel 103A.The use of different tunnels is more like to increase the probability ofdata-2-1 to reach communications router 108 successfully.

FIG. 9 is a flowchart illustrating a process carried out atcommunications router 106 according to one of the embodiments of thepresent invention. Communications router 106 receives data from host 103in step 901, where the data is destined to host 105. Communicationsrouter 106 determines, in step 902, whether the error correction mode isactivated. If the error correction mode is activated, communicationsrouter 106 sends both the original data and error correction data forthe original data in step 903. The original data is same as datareceived originally from host 103. Alternatively, if the errorcorrection mode is not activated, communications router only sends theoriginal data in step 904. The process ends in step 905.

FIG. 10 is a flowchart illustrating a process carried out atcommunications router 108 according to one of the embodiments of thepresent invention. Communications router 108 receives data fromcommunications router 106 in step 1001. Communications router 108 thendetermines in step 1002 whether all data is received on time. When errorcorrection mode is activated in communications router 106, all datacomprises original data sent by host 103, and error correction dataoriginated at communications router 106. If all data, is received ontime, communications router 108 sends a back-to-normal message tocommunications router 106 in step 1003. Communications router 108 mayforward the original data to host 105. Alternatively, if all data is notreceived on time, communications router 108 sends a request tocommunications router 106 in step 1001, where the request is to resendthe data. The process ends in step 1005.

In one variant, if the error correction mode is already deactivated,1003 is omitted. Communications router 108 determines whether the errorcorrection mode is activated or not.

It is preferred that communications router 108 sends the request toreseed data in step 1001 before host 103 determines that the data hasnot been transmitted successfully. When communications router 100receives the request to resend data, it resends a copy of the data tocommunications router 108. Communications router 106 may also send errorcorrection data to communications router 108.

Communications routers 106 and 108 preferably have a large cache memory.According to the present invention, communications router 106 has tocreate and send error correction packets and copies of packets based onpackets sent by host 103 and saved in the cache memory of communicationsrouter 106. Therefore communications router 106 may need to save manypackets in its cache memory. Additionally, a plurality of managementmessages, such as back-to-normal message and delay inquiry message mayneed to be exchanged between communications routers 106 and 108. Thesemanagement messages may also need to be saved in the cache memory ofcommunications router 106 and 108.

It should be appreciated that communications router 108 is capable ofperforming the same processes as communications router 106 and viceversa. In the above description, communications router 106 has beendescribed as the data sender and communications router 108 has beendescribed as the data receiver for readability. It should be noted thatthe scope of the invention is not limited to only host 103 sending datato host 105, such that data transmission can also take place from host105 to host 103. The data transmission can also be bidirectional, suchthat both host 103 and host 105 may send data to each other.

As communication routers need to store data for resending, storagemedium is used for storing the data. The amount of storage requiredshould be able to hold at least a few seconds of data being transmitted.Tt is preferred to store about twenty seconds of data transmission. Forexample, for an access connection that is capable of sending 100 Mbitper seconds, the amount of storage should be about twenty seconds times100 Mbit per seconds and-results in about 250 MBytes of storage. Thosewho are skilled in the art would appreciate that the larger the storageis, the more data can be retransmitted.

1. A method carried out at a first communications router fortransmitting data packets to a second communications router through aplurality of WAN interfaces: a. receiving a first data packet from afirst host through a local area network (LAN) Interface, wherein thefirst data packet is destined to a second host reachable through thesecond communications router; b. transmitting the first data packet tothe second communications router through a first WAN interface; c.storing the first data packet in a local storage medium; d.retransmitting the first data packet to the second communications routerthrough a second WAN interface; e. transmitting an error correctionpacket corresponding to the first data packet through a third WANinterface; wherein at least two of the first WAN interface, the secondWAN interface and the third WAN interface are two different WANinterfaces.
 2. The method of claim 1, wherein steps c to e are performedonly when an error correction mode is activated.
 3. The method of claim2, wherein the error correction mode is activated when the firstcommunications router receiving a delay inquiry message from the secondcommunications router.
 4. The method of claim 3, wherein the first delayinquiry message comprises a first global sequence number of the firstdata packet.
 5. The method of claim 3, wherein when a plurality of datapackets have not been received by the second communications routersuccessfully, the delay inquiry message comprises a plurality of globalsequence numbers of the plurality of data packets.
 6. The method ofclaim 1, wherein the error correction packet is further based on aplurality of data packets belonging to the same session of the firstdata packet.
 7. The method of claim 1, wherein the first data packet istransmitted through a first tunnel established between the firstcommunications router and the second communication router through thefirst WAN interface; wherein the retransmission of the first data packetis transmitted through a second tunnel established between the firstcommunications router and the second communication router through thesecond WAN interface; and wherein the error correction packet istransmitted through a first tunnel established between the firstcommunications router and the second communication router through thethird WAN interface.
 8. The method of claim 1, wherein the errorcorrection mode is activated by an administrator of the firstcommunications router.
 9. The method of claim 1, wherein each tunnel isassigned with a respective weight value based on bandwidth properties ofthe tunnel.
 10. The method of claim 9, further comprising changing therespective weight value as the packet drop rate changes.
 11. A systemfor transmitting data packet from a first communications router to asecond communications router, wherein the first communications routercomprising: at least one processing unit; at least one first computerreadable storage medium; a plurality of network interfaces; and at leastone second computer readable storage medium storing program instructionsexecutable by the at least one processing unit for: a. receiving a firstdata packet from a first host through a local area network (LAN)Interface, wherein the first data packet is destined to a second hostreachable through the second communications router; b. transmitting thefirst data packet to the second communications router through a firstWAN interface; c. storing the first data packet in a local storagemedium; d. retransmitting the first data packet to the secondcommunications router through a second WAN interface; e. transmitting anerror correction packet corresponding to the first data packet through athird WAN interface; wherein at least two of the first WAN interface,the second WAN interface and the third WAN interface are two differentWAN interfaces.
 12. The system according to claim 11, wherein steps c toe are performed only when an error correction mode is activated.
 13. Thesystem according to claim 12, wherein the error correction mode isactivated when the first communications router receiving a delay inquirymessage from the second communications router.
 14. The system accordingto claim 13, wherein the first delay inquiry message comprises a firstglobal sequence number of the first data packet.
 15. The systemaccording to claim 13, wherein when a plurality of data packets have notbeen received by the second communications router successfully, thedelay inquiry message comprises a plurality of global sequence numbersof the plurality of data packets.
 16. The system according to claim 11,wherein the error correction packet is further based on a plurality ofdata packets belonging to the same session of the first data packet. 17.The system according to claim 11, wherein the first data packet istransmitted through a first tunnel established between the firstcommunications router and the second communication router through thefirst WAN interface; wherein the retransmission of the first data packetis transmitted through a second tunnel established between the firstcommunications router and the second communication router through thesecond WAN interface; and wherein the error correction packet istransmitted through a first tunnel established between the firstcommunications router and the second communication router through thethird WAN interface.
 18. The system according to claim 11, wherein theerror correction mode is activated by an administrator of the firstcommunications router.
 19. The system according to claim 11, whereineach tunnel is assigned with a respective weight value based onbandwidth properties of the tunnel.
 20. The system according to claim19, wherein the second computer readable storage medium further storingprogram instructions executable by the at least one processing unit forchanging the respective weight value as the packet drop rate changes.